1. Field of the Invention
Embodiments of the present invention generally relate to a memory management and code exploit prevention and, more particularly, to a method and apparatus for monitoring a computer to detect operating system process manipulation by malicious software programs.
2. Description of the Related Art
In a typical computing environment, an organization may employ a number of technologies to process, store, and secure mission critical data. For example, the organization may employ one or more security systems to detect and/or mitigate network-based threats, such as malicious software programs (e.g., a virus, a Trojan and/or the like), intrusions, SPAM and/or the like. Such security systems may monitor operating system processes and exert control over a computer memory to prevent damage and/or serious software vulnerabilities within the computing environment caused by the malicious software programs.
Occasionally, the malicious software programs utilize various techniques for exploiting such software vulnerabilities in order to disrupt operations within a computer. For example, the malicious software programs may exploit a buffer overflow vulnerability and modify a stack (i.e., control stacks, call stacks and/or the like) to facilitate operation system process manipulation. The stack may refer to a special area of computer memory that is directly managed and used by processor instructions to “push” data onto the stack for storage and/or “pop” data off of the stack for retrieval. The stack may be used to store and retrieve data related to system calls (e.g., return locations and values), system call local data (e.g. local variables, including buffers), exception frames, and arbitrary data.
For instance, the malicious software programs manipulate operating system processes by injecting code into a buffer (e.g., a fixed-length portion of computer memory within the stack). As a result, the injected code overwrites an adjacent portion of the call stack that is outside a memory region allocated to the buffer and causes a buffer overflow. To emphasize, sometimes the injected code overwrites a return pointer on the call stack with a pointer to another memory region that contains executable software code (e.g., malicious software code as well as non-malicious or arbitrary software code).
Malicious software programs are not limited to code injection techniques for attacking computers. For instance, return-oriented programming is a technique by which software vulnerabilities are exploited without injecting code. Accordingly, return-oriented programming is utilized to craft buffer overflows that return to snippets of byte sequences in executable code pages to execute arbitrary system calls. New computations are constructed by linking code snippets that end with a return instruction (i.e., “ret”). Return instructions enable an attacker who controls the call stack to chain instruction sequences together. Because the executed code is marked executable in the computer memory, security systems that allow pages (e.g., pages on the stack) to be marked as non-executable (e.g., WAX and DEP) are unable to prevent execution of the chained instruction sequences. Such security systems operate on the erroneous assumption that preventing malicious code injection is sufficient to prevent the introduction of malicious computation.
Therefore, there is a need in the art for a method and apparatus for monitoring a computer to detect operating system process manipulation by malicious software programs.